Skip to content

fix crash in llama mode in llava image-to-text generation#755

Merged
regisss merged 1 commit into
mainfrom
llava_crash
Mar 4, 2024
Merged

fix crash in llama mode in llava image-to-text generation#755
regisss merged 1 commit into
mainfrom
llava_crash

Conversation

@sywangyi
Copy link
Copy Markdown
Collaborator

@sywangyi sywangyi commented Mar 4, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@sywangyi
fix crash in llama mode in llava image-to-text generation
5fd3631 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi sywangyi requested a review from a user March 4, 2024 05:28
@sywangyi
Copy link
Copy Markdown
Collaborator Author

sywangyi commented Mar 4, 2024

crash log like

------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/optimum-habana/examples/image-to-text/run_pipeline.py", line 121, in <module>
    main()
  File "/optimum-habana/examples/image-to-text/run_pipeline.py", line 111, in main
    generator(images, prompt=args.prompt, batch_size=args.batch_size, generate_kwargs=generate_kwargs)
  File "/transformers/src/transformers/pipelines/image_to_text.py", line 111, in __call__
    return super().__call__(images, **kwargs)
  File "/transformers/src/transformers/pipelines/base.py", line 1143, in __call__
    outputs = list(final_iterator)
  File "/transformers/src/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/transformers/src/transformers/pipelines/pt_utils.py", line 125, in __next__
    processed = self.infer(item, **self.params)
  File "/transformers/src/transformers/pipelines/base.py", line 1068, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/transformers/src/transformers/pipelines/image_to_text.py", line 172, in _forward
    model_outputs = self.model.generate(inputs, **model_inputs, **generate_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/optimum-habana/optimum/habana/transformers/generation/utils.py", line 798, in generate
    return self.greedy_search(
  File "/optimum-habana/optimum/habana/transformers/generation/utils.py", line 1393, in greedy_search
    outputs = self(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1521, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1530, in _call_impl
    return forward_call(*args, **kwargs)
  File "/transformers/src/transformers/models/llava/modeling_llava.py", line 469, in forward
    outputs = self.language_model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1521, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1571, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 778, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1521, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1571, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 684, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1521, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1571, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 473, in forward
    output_pre_attn, self_attn_weights, present_key_value = self.pre_attn(
  File "/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 518, in pre_attn
    output_attn, attn_weights, present_key_value = self.self_attn.pre_attn_forward(
  File "/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 266, in pre_attn_forward
    kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
AttributeError: 'tuple' object has no attribute 'get_usable_length'

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Mar 4, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

regisss
regisss approved these changes Mar 4, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@regisss regisss merged commit b9064bb into main Mar 4, 2024
@regisss regisss deleted the llava_crash branch March 4, 2024 16:51
puneeshkhanna pushed a commit to puneeshkhanna/optimum-habana-fork that referenced this pull request Mar 11, 2024
@sywangyi
Fix crash in llama mode in llava image-to-text generation (huggingfac…
e1b5f0c 
…e#755)

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024
@sywangyi @HolyFalafel
Fix crash in llama mode in llava image-to-text generation (huggingfac…
ea71b50 
…e#755)

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
gplutop7 pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Oct 15, 2025
@HabDevops @regisss
Gemma3 (huggingface#2233) (huggingface#755)
bcb881c 
Co-authored-by: Iman Gohari <s.m.iman.gohari@intel.com>
Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@regisss regisss regisss approved these changes

@mandy-li mandy-li Awaiting requested review from mandy-li mandy-li is a code owner

@libinta libinta Awaiting requested review from libinta libinta is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sywangyi @HuggingFaceDocBuilderDev @regisss